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ABSTRACT 

Synthetic vision is a computer-generated image of the external scene topography that is generated from aircraft attitude, 
high-precision navigation information, and data of the terrain, obstacles, cultural features, and other required flight 
information. A synthetic vision system (SVS) enhances this basic functionality with real-time integrity to ensure the 
validity of the databases, perform obstacle detection and independent navigation accuracy verification, and provide 
traffic surveillance. Over the last five years, NASA and its industry partners have developed and deployed SVS 
technologies for commercial, business, and general aviation aircraft which have been shown to provide significant 
improvements in terrain awareness and reductions in the potential for Controlled-Flight-Into-Terrain incidents / 
accidents compared to current generation cockpit technologies. 

It has been hypothesized that SVS displays can greatly improve the safety and operational flexibility of flight in 
Instrument Meteorological Conditions (IMC) to a level comparable to clear-day Visual Meteorological Conditions 
(VMC), regardless of actual weather conditions or time of day. An experiment was conducted to evaluate SVS and 
SVS-related technologies as well as the influence of where the information is provided to the pilot (e.g., on a Head-Up or 
Head-Down Display) for consideration in defining landing minima based upon aircraft and airport equipage. The 
“operational considerations” evaluated under this effort included reduced visibility, decision altitudes, and airport 
equipage requirements, such as approach lighting systems, for SVS-equipped aircraft. Subjective results from the 
present study suggest that synthetic vision imagery on both head-up and head-down displays may offer benefits in 
situation awareness; workload; and approach and landing performance in the visibility levels, approach lighting systems, 
and decision altitudes tested. 

Keywords: Synthetic Vision System, Head-Up Display, Global Positioning System, Aviation Safety, Next Generation 
Air Transportation System, Approach Lighting System 

1. INTRODUCTION 

The Integrated Intelligent Flight Deck Technologies (IIFDT) project, under NASA’s Aviation Safety Program (AvSP), 
comprises a multi-disciplinary research effort to develop flight deck technologies that mitigate operator-, automation-, 
and environment-induced hazards. Towards this objective, IIFDT is developing crew/vehicle interface technologies that 
reduce the risk of pilot error, improve aircraft safety for current and future civilian and military aircraft, and proactively 
overcome aircraft safety barriers that would otherwise constrain the full realization of the Next Generation Air 
Transportation System (NGATS). Part of this research effort involves the use of enhanced and synthetic vision systems 
and other interface modalities as enabling technologies to meet the challenges of an Equivalent Visual Operations 
(EVO) concept - that is, the ability to achieve or even improve on the safety of Visual Flight Rules (VFR) operations, 
maintain the operational tempos of VFR, and even, perhaps, retain VFR procedures independent of actual weather and 
visibility conditions. 

One significant challenge to the EVO concept objective is the definition of required equipage on the aircraft and on the 
airport. With today’s equipment and regulations, significant investment is required in on-board equipment for 
navigation, surveillance, and flight control and on the airport for lighting systems for “all-weather” landing capability. 
The levels of equipment redundancy, capability, and accuracy dramatically increase as landing visibility minima 
decrease. A vivid demonstration of the current complexity (and cost) to meet “weather and visibility-independent” 
capability is to look at the present National Air Space (NAS) infrastructure. There are approximately 2,600 Instrument 
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Landing System (ILS) approaches to Category 1 minima (no lower than 200 ft decision height, 2400 ft visibility) 
available throughout the United States, but only 211 Category II (no lower than 100 ft decision height, 1200 ft visibility) 
and 126 Category III (a decision height lower than 100 ft, or no decision height, or a runway visual range (RVR) less 
than 1200 ft) approaches. EVO implies “Category III” operations to all runway thresholds. As previously mentioned, 
synthetic vision (SV) displays may be one of the crew/vehicle interface technologies that help achieve the realization of 
EVO in the next generation NAS. 

Synthetic vision is a computer-generated image of the external scene topography that is generated from aircraft attitude, 
high-precision navigation, and data of the terrain, obstacles, cultural features, and other required flight information (Fig. 
1). A synthetic vision system (SVS) enhances this basic functionality with real-time integrity to ensure the validity of 
the databases, perform obstacle detection and independent navigation accuracy verification, and provide traffic 
surveillance. Over the last five years, NASA and its industry partners have developed and deployed SVS technologies 
for commercial and business aircraft which have been shown to provide significant improvements in terrain awareness 
and reductions for the potential of Controlled-Flight-Into-Terrain incidents/accidents compared to current generation 
cockpit technologies. 1 ' 6 

It has been hypothesized that the use of Synthetic Vision technologies on head-up and head-down displays can provide 
precision approach, landing, and taxi guidance for “all weather” capability to all runways without perhaps, requiring 
extensive approach lighting systems, ground-based precision guidance systems such as the Instrument Landing System 
(ILS), or other airport infrastructure. An experiment was conducted to identify the influence of cockpit display 
information (SV vs. Conventional) and airport lighting information and where the cockpit display information is 
provided to the pilot (e.g., on a Head-Up or Head-Down Display) on landing minima. A simulated Global Positioning 
System/Wide Area Augmentation System (GPSAVAAS) was used in the experiment for precise approach guidance, 
instead of an ILS, and for the navigation component of the on-board SVS. The “operational considerations” evaluated 
under this effort included reduced visibility, decision altitudes, and airport approach lighting systems, for SVS-equipped 
and non-equipped aircraft. 
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Figure 1: Synthetic Vision System 


2 



2. METHODOLOGY 


2.1. Subjects 

Twenty-three pilots, representing seven airlines and the Federal Aviation Administration (FAA) Flight Standards and 
Transport Aircraft Certification Branches, participated in the experiment. All but three participants had previous 
experience flying Head-Up Displays (HUDs). The subjects had an average of 1231 hours of HUD flying experience and 
an average of 16.8 years and 14.2 years of commercial and military flying experience, respectively. 

2.2. Simulation Facility 

The experiment was conducted in the Integration Flight Deck (IFD) simulation facility (see Fig. 2) at NASA Langley 
Research Center (LaRC). The IFD emulates a Boeing B-757-200 aircraft and provides researchers with a full-mission 
simulator capability. The cab is populated with flight instrumentation and pilot controls, including the overhead 
subsystem panels, to replicate the B-757 aircraft. The collimated out-the-window (OTW) scene is produced by an Evans 
and Sutherland ESIG 4530 graphics system providing approximately 200 degrees horizontal by 40 degrees vertical field- 
of-view at 26 pixels per degree. 



Figure 2. Integration Flight Deck Simulation Facility with HUD and Head-Down Research Display (HD-RD). 

The evaluation pilot (EP) occupied the left seat, as the Pilot Flying (PF) for this experiment. The left seat included an 
overhead HUD projection unit and a head-down research display (RD) (see Fig. 2). 

2.2.1. Head-Up Display 

The HUD subtended approximately 32° horizontal by 24° vertical field of view (FOV). Note that to maintain 
conformality with the outside world, the FOV for the HUD SVS (raster) image was fixed and could not be varied by the 
EP. The HUD presentation was written strictly in raster format from a video source (RS-343) input. The input consisted 
of a video mix of symbology and computer-generated scene imagery. The symbology included “haloing” to ensure that 
the symbology was highlighted against the scene imagery background. Overall HUD brightness and contrast controls 
were provided to the pilot. In addition, the EP was able to independently adjust the flight symbology brightness relative 
to the SVS raster. The pilot also had a declutter control, implemented as a push-button on the left hand hom of the PF 
yoke. The button cycled through three “declutter” states: 1) No declutter (full symbology and scene imagery); 2) 
“Raster” declutter (full symbology, no scene imagery); and 3) “Full declutter” (no HUD display). 

2.2.2. Head-Down Research Display 

A head-down research display (RD) was installed over the normal instruments on the left hand side of the IFD cockpit 
(see Fig 2). This implementation approach was chosen for convenience and cost. The RD used an 18.1 -inch diagonal 
case containing two high brightness liquid crystal displays (LCDs). The two separate LCD panels, each with XGA 
(1024x768) resolution, rendered a Size D (6.4 inch square viewable area) primary flight display, PFD (left display), and 
navigation display, ND (right display). The RD covered the normal Boeing 757 captain’s displays with the exception of 
the analog standby instruments (attitude direction indicator, airspeed, and altitude). 
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2.2.3. Head-Down Displays 

The PFD and ND closely resembled current transport aircraft equipage. The PFD was only modified from typical 
equipage to include a Flight Path Marker (FPM) and guidance cue. The PFD FPM and guidance cue were driven by 
algorithms identical to the HUD. Standard B-757 ship’s flight director needles were disabled. 

The ND showed the Dallas/Ft. Worth (DFW) International Airport Runway 18 Right (18R) approach path, but it did not 
include any Enhanced Ground Proximity Warning System nor Traffic Alert and Collision Avoidance System (TCAS) 
information. 

2.3. Synthetic Vision Database 

A synthetic vision database was created from a 0.33 arcsec (-10 meter post-spacing) Digital Elevation Model (DEM) of 
a 53 x 57 nm area centered around the Dallas- Ft. Worth International Airport (FAA identifier: KDFW). The DEM was 
draped with elevation-based generic imagery. The airport was represented by three-dimension models of the runway, 
taxiways, and terminal buildings which were extracted from aerial photography of approximately 1 foot resolution. 

2.4. Symbology 

2.4.1. HUD Symbology 

The HUD stroke symbology format was loosely based on the HGS-4000 “Primary Mode” stroke symbology set, with the 
compass rose symbol removed (see Fig. 3). Glideslope and localizer raw data indicators which included a deviation 
scale and angular deviation indication were provided (i.e., glideslope and localizer deviation). The altitude tape included 
a horizontal bar representing the current Decision Altitude (DA), and a barber-pole presentation to indicate field 
elevation. 

The pitch-roll guidance cue (“ball”) used modified pursuit guidance 7 along the desired path centerline, 5.5 seconds ahead 
of ownship. Horizontal and vertical position of the ball reflects the track and flight path angles to fly to the center of the 
desired path. The path deviation indicators showed angular course deviation (i.e., glideslope and localizer-like) 
conditions by converting the linear path error data to angle errors and scaling in “dots.” 



Figure 3. HUD Symbology 

A glideslope reference line was drawn (Fig. 4) at the DFW Runway 18R Instrument Landing System (ILS) descent angle 
of 3.0 degrees. Also, a runway outline symbol was drawn using the threshold coordinates of the DFW 18R/36L runway 
based on the simulated aircraft navigation solution to conformally position the symbol. The runway outline was drawn 
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using an 8000 ft x 200 ft runway. Also, radar altitude was shown digitally underneath the FPM when below 500 ft 
above ground level (AGL). . 



Figure 4. HUD Runway Outline Symbol and Glideslope Reference Line. 

2.4.2. PFD Symbology 

The primary differences between the PFD symbology (Fig. 5) and the HUD symbology were that the PFD had an alpha- 
numeric readout of the current DA, the radar altitude was displayed above the altitude tape instead of below, and the 
vertical speed was represented by a trend arrow instead of a numeric readout. Like the HUD, radar altitude was digitally 
displayed under the FPM when below 500 ft AGL. 



Figure 5. PFD Symbology 


Like the HUD, a glideslope reference line was drawn on the PFD at the DFW Runway 18R ILS descent angle of 3.0 
degrees. Also, a runway outline symbol was drawn using the threshold coordinates of the DFW 18R/36L runway based 
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on the simulated aircraft navigation solution to conformally position the symbol. Unlike the HUD, the runway outline 
on the PFD (Fig. 6) was only drawn when the PFD background was synthetic vision, and the runway outline was drawn 
conformal in size to the selected runway and included an extended center line. For DFW 18R, the runway outline was 
150 feet wide and 11388 feet long and the extended runway centerline was 3000 feet long. 



Figure 6. PFD Runway Outline Configuration 


2.5. Display Concepts 

Four display concepts (baseline and SV, either head-up or head-down) were evaluated by the evaluation pilots while 
flying approaches to DFW Runway 1 8R. The head-down navigation display format was invariant. 

2.5.1. Head-Up Display Concepts 

Two HUD display concepts were tested, differing from each other only in the type of raster background (SV or none) 
presented. Standard HUD symbology enhanced with a runway outline was employed in both HUD concepts. In Figure 
7, the two HUD Concepts are shown - the Baseline HUD (left) and the SV HUD (right). During the HUD experimental 
runs, the “baseline” PFD was displayed (PFD format shown on the left side of Figure 8.). 






H 

F6Eh 


tna 

— 

Ivw** 

TUMP — 


U J-£iU 
□ [ 

■d 

“ — ■ 


TSEU — 

■ " t “ * 3 

■* KCE 

,L 

iaa 


- eo4 

m 

* 

■ I HU 

_ . 

:* HI 1 h 


, \P JP © _ _ 

JIT -Ml 



PT3MI 



Figure 7. Head-Up Display (HUD) Formats - Baseline (left) and Synthetic Vision (right). 
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2.5.2. Head-Down Display Concepts 

Two PFD display concepts were tested, differing from each other only in the type of background (standard sky/ground or 
SV) presented with the flight symbology. Figure 8 presents the two PFD Concepts - the Baseline PFD (left) and the 
Synthetic Vision PFD (right). For the head-down display evaluations, the HUD was stowed to preclude blocking or 
distortion of the pilot’s forward view of the outside world. 



Figure 8. Primary Flight Display (PFD) Formats - Baseline (left) and Synthetic Vision (right). 


2.6. Approach Lighting System Configurations 

Three different approach lighting system (ALS) configurations were used for this experiment as shown in Figure 9. The 
first ALS configuration (hereinafter referred to as the VFR ALS) was representative of lights found at a VFR runway 
and included Runway Edge Identification Lights (REIL), precision approach path indicator lights (PAPI), partial 
threshold lights, and medium intensity runway lights (MIRL). The second ALS configuration (hereinafter referred to as 
the MALSR ALS) was representative of a Category I/Type I operations runway and included MALSR (medium 
intensity approach lighting system; runway alignment indicator lights), PAPI, full threshold lights, and MIRL. The third 
ALS configuration (hereinafter referred to as the ALSF-2 ALS) was representative of a standard Category II/Type II 
operations runway and included ALSF-2 (high intensity approach lighting system with sequenced flashing lights), 
touchdown zone and centerline lighting, PAPI, full threshold lights, and high intensity runway lights (HIRL). 

2.7. Evaluation Task 

The evaluation task was a straight- in GPS WAAS-type approach with a 3 degree descent angle to Runway 18R at DFW 
airport. For each run, the approach started 5 nautical miles (nm) from the runway threshold and the aircraft was placed 
one dot left or right of the course centerline, and one dot high or low of the course glidepath. The weather consisted of 
altitude-based cross winds (wind direction and intensity was dependent on altitude), light turbulence (root-mean- square 
(rms) of 2 ft/sec), and varying visibility levels (3 miles, 2400 ft, 1800 ft, or 1200 ft RVR). The wind profile started out 
as a 20 knot left or right quartering head- wind and reduced to a direct 10 knot cross-wind at field elevation. The 
evaluation pilot hand- flew the approach from the left seat with auto-throttles engaged at a speed of 138 knots. The run 
was terminated at full-stop or upon go-around initiation. The aircraft was configured to land prior to each run (landing 
gear down and flaps 30 degrees), and the aircraft was “Cleared to Land.” There were no other aircraft in the 
environment, and there was no ATC involvement in the task. 

Each approach had a DA of either 200 ft or 100 ft. For the 200 ft DA approaches, the evaluation pilot was required to 
visually acquire the approach lighting system by 200 ft above the Touchdown Zone Elevation (TDZE) or a go-around 
was required. If the approach lighting system was visually acquired by 200 ft above the TDZE, then the pilot was 
cleared to continue the approach but he/she could not descend below 100 ft above the TDZE unless the runway 
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environment was distinctly visible and identifiable. For the 100 ft DA approaches, the evaluation pilot was required to 
visually acquire the runway environment by 100 ft above the TDZE to land. Otherwise, a missed approach was 
executed. 



Figure 9. Approach Lighting System Configurations - VFR (left), MALSR (center), and ALSF-2 (right) 

The evaluation pilots were instructed to fly the aircraft as if there were passengers aboard, fly the center of the approach 
path, and land as close as possible to the centerline and touchdown zone. After landing, they were to capture the center 
line and come to a complete stop on the runway. They were also instructed to initiate a go-around if the landing was not 
safe or there were any safety concerns during the approach. 

Prior to run commencement, the EP was briefed on the initial starting position (either 1 dot left or right of course and 
either 1 dot high or low of course), the display concept to be evaluated (Baseline HDD, SV HDD, Baseline HUD, or SV 
HUD), Decision Altitude (100 or 200 ft), and the wind direction (left or right crosswind). The visibility level was not 
briefed to the pilot prior to run initiation. 

2.8. Pilot Procedures 

Since only pilot-flying evaluations were being conducted, an automatic aural call-out was included in the simulation to 
“assist” in altitude awareness. A “500 feet” call-out was made at 500 ft above field level (AFL). An “Approaching 
Minimums” call-out was made at 100 ft above the DA for the run. Finally, a “Minimums” call-out was made at the pre- 
set DA. The altitude call-outs were set-up assuming a decision altitude based on barometric pressure. The field 
elevation for DFW was 607 ft. Hence, for a DA of 100 feet, the aural callout for “minimums” occurred with a 
barometric altimeter reading of 707 ft MSL. 

The EPs were instructed to verbally denote and acknowledge their recognition of the required runway visual references. 
For approaches where the DA was 200 ft, there was an automatic call-out of “Approaching Minimums” at 300 ft AFL 
and an automatic call-out of “Minimums” at 200 ft AFL. The pilot had to visually acquire the approach lighting system 
by 200 ft AFL to continue the approach. The pilot was instructed to call “Lights” when the approach lighting system 
became visible. If this call was made before descending to 200 ft AFL, the approach could continue, else a call of “go- 
around” was required and the run was terminated. If the approach continued, the pilot was instructed to call “Landing” 
when the required landing visual references (using natural vision) as specified in Federal Aviation Regulations (FAR) 
Part 91.175c became identifiable. If this call was made before 100 ft AFL, the approach could continue to landing, else 
a call of “go-around” was required and the run was terminated. 
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For approaches where the DA was 100 ft, there was an automatic call-out of “Approaching Minimums” at 200 ft AFL 
and an automatic call-out of “Minimums” at 100 ft AFL. The pilot was instructed to call “Lights” when the approach 
lighting system became visible and to call “Landing” when the required landing visual references as specified in FAR 
Part 91.175c became identifiable. If the “Landing” call was made before 100 ft AFL, the approach could continue to 
landing, else a call of “go-around” was required and the run was terminated. 

The evaluation pilots were allowed to continue the approach and to land at their own discretion. If the “Landing” call 
was not made by 100 ft AFL, or if the “Lights” call was not made by 200 ft AFL for approaches with a 200 ft DA, the 
approach was not automatically terminated. This protocol was used to assess the effect of the experimental conditions 
(display type, approach lighting system, visibility) on the pilot’s altitude awareness. 

Field use of this technology could be in a two-crew aircraft. Since the experimental setup for this evaluation used single 
crew, the automatic aural call-outs were provided in part to simulate the functionality of the pilot monitoring. The 
“Approaching Minimums” and “Minimums” call-outs represent the least amount of functionality typically provided by 
the monitoring pilot. If this technology were introduced in a single-crew cockpit, the test procedure assumes the 
presence of an automatic call-out system. Hence, this test does not directly address single-pilot operations without 
altitude alerting on the approach, although one of the non-normal conditions tested (as detailed in the following section) 
does address this situation. 

2.9. Experiment Matrix 

Nominally, ten training runs and fifty- three experimental runs were completed by the EP. 

The primary experiment matrix consisted of combinations (but not a full-factorial) of Display Concept (Baseline HDD, 
SV HDD, Baseline HUD, or SV HUD), ALS (VFR, MALSR, or ALSF-2), runway visibility range (1200 ft, 1800 ft, 
2400 ft, or 3 statute miles), and DA (100 or 200 ft) as shown in Table 1. Two additional runs (one at 2400 ft RVR/ 200 
ft DA/ MALSR ALS and one at 3 mile RVR/ VFR ALS) using the Baseline HDD concept but without a flight director 
were also completed by each EP. 


Table 1: Primary Experiment Matrix 
(Evaluated Using Each of the Four Display Concepts) 


Visibility Decision Altitude 

VFR ALS 

MALSR 

ALSF-2 

1200 ft RVR 100 ft 

V 

V 

v 

200 ft 


V 


1800 ft RVR 100 ft 


V 


200 ft 


< 


2400 ft RVR 100 ft 

■V 

< 

V 

200 ft 


V 


3 statute miles n/a 

V 




A significant component of the test, in addition to the nominal runs, was the investigation of the ability of the EP to 
recognize and properly handle non-normal events. Seven non-normal runs were flown by each EP. The non-normals 
were four database integrity monitoring scenarios, two altimetry failure scenarios, and one SV obstacle placement error 
scenario. Each of the non-normal runs used the MALSR ALS with a 200 ft DA and 2400 ft RVR. The number of non- 
normal scenarios was designed to avoid expectancy on the part of the flight crew. 8 The database integrity monitoring 
scenarios purposefully introduced either a lateral navigation solution error (of 40 meters) or an altitude navigation 
solution error (50 feet low) with respect to the real runway. This error resulted in the synthetic vision terrain and 
guidance cue being misaligned from the ALS and actual runway environment. The altimetry failure scenario emulated a 
failure inhibiting the aural callouts and the display of radar altitude information and flare cue symbology to the pilot. 
The intent of this scenario was to assess the impact of the aural callouts and radar altitude display on the pilot’s ability to 
be cognizant of the DA, make the appropriate pilot callouts (e.g., “lights”, “landing” or “go-around”), and perform a safe 
approach and landing. In the SV obstacle placement error scenario, a water tower which was located 2 miles from the 
runway threshold in the OTW view (present on every run), had its SV icon (representing the obstacle) erroneously 
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placed only 1 mile from the runway threshold. This scenario results in a mismatch of the SV imagery with the OTW 
scene for the water tower obstacle and presents a prominent SV-depicted obstacle along the aircraft’s flight path. 

2.10. Measures 

During each run, path error, pilot control inputs, and touchdown performance (sink rate at touchdown, distance for or aft 
of touchdown zone, and distance left or right of centerline) were recorded for analysis. 

After each run, pilots completed a run questionnaire consisting of the NASA Task Load Index (TLX) workload rating 9 , 
Situation Awareness Rating Technique (SART) 9 , and six Likert-type (5-point) questions specific to different constructs 
of making a stabilized and safe approach (see Fig. 10). Question 1-4 in Figure 10 were identical to those used in an FAA 
study 10 evaluating the effects of reduced approach lighting systems on the minimum pilot visual cues needed during an 
approach at Category I minimums (i.e., MALSR ALS used in the present experiment). 


Post-Run Approach 
Ratings 

Please rate your: 

Excellent 

1 

Good 

2 

Acceptable 

3 

Almost 

Acceptable 

4 

Absolutely 

Unacceptable 

5 

Q1 . Ability in finding the 
runway and touchdown zone 






Q2. Awareness of altitude 
above the ground 






Q3. Lateral alignment with 
the runway 






Q4. Roll guidance 






Q5. Ability to pick up the 
necessary visual references 
after transitioning from the 
displays to out the window 






Q6. Ability to complete the 
approach and land safely 







Figure 10. Post-Run Approach Questions 


After data collection was completed, pilots were administered the Situation Awareness - Subjective Workload 
Dominance (SA-SWORD) 11 and Subjective Workload Dominance (SWORD) 9 tests for the display concept (Baseline 
HDD, SV HDD, Baseline HUD, SV HUD) comparisons. The pilots also completed a post-test questionnaire to elicit 
comments on the display concepts with the various approach lighting systems, the visual transition strategy employed by 
the pilot with the different display concepts, the pilot procedures, and the fidelity of the simulator. 

For the post-run questions, analysis of variance (ANOVA) analyses were conducted for the factors of display location 
(HDD, HUD), display information content (Baseline, SV), ALS (VFR, MALSR, ALSF-2), RVR (1200, 1800, 2400), 
and DA (100, 200). In addition, the EP background (i.e., “user” - transport pilot or “regulators” - FAA pilot) was also 
used as a factor since the pilot groups may have significant differences in viewpoint, experiences, currency, and number 
of flight hours. When the 2 nd order interaction was significant, a simple main effects analysis was conducted with the 
significance level of alpha (a) set at 0.05. For the post-test paired comparisons, ANOVAs were conducted for the factors 
of display concept (Baseline HDD, SV HDD, Baseline HUD, SV HUD) and pilot work experience (transport pilot, FAA 
pilot). When necessary, Student-Newman-Keuls (SNK) post-hoc tests with a set at 0.05 were performed. 

2.11. Test Conduct 

The subjects were given a 1-hour briefing to explain the HUD and PFD concepts (baseline and SV), pilot procedures, 
and the evaluation tasks. After the briefing, a 1-hour training session in the IFD was conducted to familiarize the 
subjects with the aircraft handling qualities, display symbologies, pilot procedures, and controls. The ‘rare-event’ (or 
non-normal) scenarios were not discussed, although the pilot’s responsibility for maintaining safe operations at all times 
was stressed. Data collection lasted approximately 4.5 hours and was followed with the application of the S WORD/S A- 
SWORD paired comparisons technique and a final questionnaire. The entire session including lunch and breaks lasted 
approximately 1.25 days. 
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3. RESULTS 

At the time of this publication, the objective data analyses have not been completed. Results are presented for the 
subjective analyses. 

3.1. Mental Workload 

Mental workload was assessed after each experimental run, using the NASA TLX, and post-test, using the SWORD 
technique. NASA TLX is a multi-dimensional rating procedure that derives an overall workload score based on a 
weighted average of ratings on six subscales (mental demand, physical demand, temporal demand, performance, effort, 
and frustration level). The NASA TLX subscales went from 0 (Low) to 100 (High) for the workload ratings. Pilots 
were administered the paired-comparison SWORD scale that enabled ratings of mental workload across the four display 
concepts (location by content) flown by the EP. The definition of mental workload used was “The amount of cognitive 
resources available to perform a task and the difficulty of that task.” 

3.1.1. NASA TLX 

The main factors, RVR (F(2, 896)=5.51, p=0.004) and experience (F(l, 896)=84.72, pO.OOl), and the interaction 
between location and content (F(l, 896)=4.34, p=0.038) were significant for workload. The other main factors of 
location (HDD, HUD), ALS (VFR, MALSR, ALSF-2), and DA (100, 200 ft) were not significant (p>0.05) for this 
measure. Post-hoc tests (SNK using a=0.05) showed two unique subsets for workload ratings with the 3 RVR values: 
1) 2400 ft (mean = 43) and 1800 ft (mean = 44) and 2) 1200 ft (mean = 48). In addition, transport (“user”) pilots had 
lower workload ratings (mean = 44) compared to FAA pilots (mean = 54). A simple main effects analysis revealed that 
the effects of information content were significant only when presented in the head-up display location. The inclusion of 
SV imagery on the HUD significantly reduced the pilots’ workload ratings compared to just the baseline symbology 
alone at that display location. 

3.1.2. SWORD 

The post-test SWORD data indicate that display concept (F(3, 84)=48.00, pO.OOl) was highly significant for the pilot 
ratings of mental workload. Post-hoc tests (SNK using a=.05) showed three unique subsets for the mental workload 
ratings with the 4 display concepts: 1) SV HUD (lowest workload); 2) SV HDD and Baseline HUD; and 3) Baseline 
HDD (highest workload). 

3.1.3. Workload Discussion 

The subjective post-test ratings indicated the workload was significantly reduced when using the HUD compared to the 
head-down displays and that the no-SV HUD workload was comparable to when SV is added to the head-down display. 
The post-run ratings indicated that the inclusion of SV imagery on the HUD significantly reduced the pilots’ workload 
ratings compared to the baseline symbology at that display location. There were no appreciable differences in workload 
ratings for the 2400 ft (Standard Cat I approach) and 1800 ft RVR conditions. Even though statistically significant 
differences between these 2 RVR values and the 1200 ft RVR were found, operationally these differences were not 
substantial - all ranked at a moderate workload level on the TLX scale. Interestingly, the commercial transport pilots 
rated their workload significantly lower than the FAA pilots, yet the data trends in the pilot workload ratings indicate 
that the FAA pilots (“regulators”) concurred with the “user” community (albeit to a lesser degree) on the workload 
benefits of SV and the HUD for low visibility operations. 

3.2. Situation Awareness 

Situation awareness was assessed after each experimental run, using the post-run SART, and post-test, using the SA- 
SWORD measures. SART is a multi-dimensional rating technique using the constructs of: 1) demand on attentional 
resources; 2) supply of attentional resources; and, 3) understanding. From these components, the SART rating is 
“understanding” reduced by the difference of “demand” minus “supply” (i.e., SART = {(understanding) - (demand - 
supply)}). Similar to the SWORD described above, the S A- SWORD is a paired-comparison technique that provides 
relative situation awareness ratings across the four display concepts (Baseline HDD, SV HDD, Baseline HUD, SV 
HUD) flown by the EP. For these comparisons, SA was defined as “The pilot’s awareness and understanding of all 
factors that will contribute to the safe flying of their aircraft under normal and non-normal conditions.” The SA- 
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SWORD measure differs from the post-run SART measure construct as it is a pair-comparison test, it was administered 
post-test, and the underlying definition and construct for the SA ratings are different. 

3.2.1. SART 

An ANOVA revealed that location (F(l, 833)=3.84, p=0.05), content (F(l, 833)=5.47, p=0.02), RVR (F(2, 833)=5.08, 
p=0.006), and Experience (F(l, 833)=74.98, p<0.001) were significant for pilot SART ratings. The main factors of ALS 
and DA and second-order interactions were not significant (p>0.05) for this measure. Pilots rated their SA significantly 
higher: 1) when using the HUD compared to the HDD and 2) when using SV information compared to the Baseline 
information set. Post-hoc tests (SNK using a=0.05) revealed 2 unique subsets for SA ratings with the 3 RVR values: 1) 
1200 - lowest SA and 2) 1800 and 2400 ft - highest SA. In addition, transport pilots gave higher SA ratings than the 
FAA pilots. 

3.2.2. SA-SWORD 

An ANOVA revealed that display concept (F(3, 84), p<0.001) was highly significant for the SA-SWORD ratings. Post- 
hoc tests (SNK using a=0.05) showed four unique subsets for situation awareness ratings with the 4 display concepts: 1) 
Baseline HDD (lowest SA); 2) Baseline HUD; 3) SV HDD; and 4) SV HUD (highest SA). Pilots ranked the SV HUD 
concept as having significantly higher SA than the other 3 display concepts tested. 

3.2.3. Situation Awareness Discussion 

Both post-run and post-test subjective measures indicate that significant SA gains were provided by the SV imagery 
compared to the Baseline condition and that the SV HUD provided the highest SA among the 4 concepts tested. Pilot 
commentary noted that the presence of SV imagery gave the pilots a much better understanding and appreciation 
(improved SA) of the approach path and runway environment. Without the imagery, pilots commented that they had to 
“blindly follow” the guidance cue (i.e., compensatory guidance symbol only) as they performed the approach. With the 
imagery, pilots commented that they understood the guidance cue movements due to the SA gains provided by it and the 
flight path marker against the SV scene. In addition, the pilots felt that this increase in SA also allowed them to make 
less aggressive maneuvers (thus improving flight path control performance) as they performed the approach. The data 
indicated that the commercial transport pilots rated their situation awareness significantly higher than the FAA pilots. 
Again, data trends in the pilot SA ratings indicate that both pilot groups agreed with the SA benefits of HUDs and SV, 
albeit in differing magnitudes. 


3.3. Landing Performance 

One important influence in setting the landing minima is the likelihood that an approach will be taken safely to landing. 
The percentage of landings flown to a landing is plotted in Figure 11. The data is presented as a function of visibility 
(RVR) with differentiation between combinations of approach lighting system type and Decision Altitude (DA). 

As shown in Figure 11, almost all of the approaches flown in 2400 ft RVR visibility, independent of the ALS, were 
flown to touchdown. Even with the VFR lighting condition, 96% (88 landings from 92 approaches) of the approaches 
ended with a landing. 

As the visibility was reduced, the percentage of landings was only minimally reduced when the MALSR (85%) or 
ALSF-2 (88%) approach lighting system was used, with a 100 ft DA. With VFR lighting, however, it was a 50-50 
probability that the approach would be completed. The EPs, even with a 100 ft DA, didn’t necessarily have the required 
landing references to complete the approach to landing. 

The percentage of approaches concluded with a landing is reduced if a DA of 200 ft is used as opposed to 100 ft DA 
(with the MALSR ALS). By 200 ft DA, the EP must be able to see the ALS to continue the approach to 100 ft Height 
Above Touchdown wherein the EP must see the landing threshold lights and/or touchdown zone to land. With 2400 ft 
RVR, the reduction in the percentage of successful approaches (using a 200 ft DA compared to a 100 ft DA) is only 
about 5%, but at 1200 ft RVR, the difference is 21%. (At 1200 ft RVR, only 59 out of 92 approaches were taken to 
landing (64%)). 
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Figure 11: Percentage of Approaches Flown to Landing. 


3.4. Post-Run Approach Assessments 

The EPs were asked six questions (Figure 10) after each run to obtain subjective feedback on their ability to: find the 
runway and touchdown zone references (Question 1); assess their altitude awareness (Question 2); understand their 
lateral position with respect to the runway and correct back to the centerline, if offset (Questions 3 and 4); and finally, 
identify the necessary visual references when transitioning from instrument to visual flight (Question 5) and safely 
executing the landing (Question 6). A Likert-type scale was used for the pilot ratings. Separate ANOVAs were 
conducted on the post-run questionnaire pilot ratings for the normal runs which ended in a landing (i.e., excluded non- 
normal runs and runs that ended in a go-around). 

The data showed trends toward improvement across all approach assessment questions for the presence of SVS instead 
of the baseline concepts (i.e., no-SVS) and using the HUD instead of the HDDs. In addition for each post-run question, 
differentiation by pilot experience (transport pilots, FAA pilots) was statistically, but not operationally, significant as 
both groups had mean ratings in the good to very good range. 

ANOVAs and subsequent post-hoc tests showed statistically significant differences for the following main factors: 

• In response to Question 1, three unique subsets showed that a 2400 ft RVR (mean rating of 1.51) provided the 
pilots with a significantly better ability to find the runway and touchdown zone, compared to a 1800 ft RVR 
(mean rating of 1.64) and 1200 ft RVR (mean rating of 1.88). Similarly, two unique subsets showed that the 
ALSF-2 and MALSR lighting configurations (mean ratings of 1.54 and 1.64, respectively) provided 
significantly better ability to find the runway and touchdown zone, compared to the VFR ALS (mean rating of 
1.88). The ALSF-2 was not statistically different than the MALSR ALS. Pilot ratings also indicated that the 
pilot’s ability to find the runway and touchdown zone references was significantly better with the HUD (mean 
rating of 1.53) than the HDD (mean rating of 1.80). 

• Except for pilot experience, there were no other significant differences in response to Question 2 - “Awareness 
of altitude above the ground”. 

• In responses to Question 3 and 4, pilot ratings indicated that the lateral alignment and roll guidance were 
significantly better with the HUD (mean rating of 1.49 for lateral alignment and mean rating of 1.64 for roll 
guidance) than the HDD (mean rating of 1.75 for lateral alignment and mean rating of 1.84 for roll guidance). 
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In addition, pilot ratings for understanding their lateral position with respect to the runway and correcting back 
to the centerline (i.e. roll guidance) were significantly better with the SVS information content (mean rating of 
1.50 for lateral alignment and mean rating of 1.61 for roll guidance) than with the Baseline condition (mean 
rating of 1.72 for lateral alignment and mean rating of 1.85 for roll guidance). 

• In response to Question 5, three unique subsets showed with a 2400 ft RVR (mean rating of 1 .70) providing the 
pilots with significantly better ability to transition from instrument to visual flight and finding the required 
visual references for landing, compared to 1800 ft RVR (mean rating of 1.86) and 1200 ft RVR (mean rating of 
2.15). Similarly, three unique subsets showed that the ALSF-2 ALS (mean ratings of 1.71) provided 
significantly better ability to transition from instrument to visual flight and finding the required visual 
references for landing, compared to the MALSR ALS (mean rating of 1.87) and the VFR ALS (mean rating of 
2.14). Pilot ratings also indicated that location (HUD mean rating of 1.79 and HDD mean rating of 1.98) and 
DA (100 ft mean rating of 1.85 and 200 ft mean rating of 1.95) were significant for Question 5. 

• In response to Question 6, two unique subsets were shown with a 2400 ft and 1800 ft RVR (mean rating of 1.63 
and 1.74, respectively) providing the pilots with significantly better ability to complete the approach and 
landing safely, compared to 1200 ft RVR (mean rating of 1.98). Pilot ratings also indicated that location (HUD 
mean rating of 1.63 and HDD mean rating of 1.92) was significant for successfully completing the approach 
and landing. 

These data indicate that the pilots felt that the cockpit displays (location and content) were influential in understanding 
their lateral position with respect to the runway and correcting back to the centerline. The SVS imagery afforded them a 
better understanding of their lateral alignment with the runway and their roll guidance information than the Baseline 
information content (no SVS imagery). The pilots also rated their understanding of this information to be higher when it 
was presented on the HUD versus on the HDD. The HUD also allowed the pilots an easier transition from the 
instrument segment to the visual segment of the approach and an improved ability in finding the runway and touchdown 
zone and in landing safely compared to the HDD location. The airport lighting and visibility conditions statistically 
affected the pilot’s ability to find the runway and touchdown zone references and to identify the necessary visual 
references when transitioning from instrument to visual flight and safety executing the landing. While these differences 
are statistically significant, even with RVR as low as 1200 ft, the pilot’s felt, on average, that they could safely complete 
the approach and landing with the display configurations and lighting tested. 

In Figure 12, the mean rating and +/- one standard deviation for the EP responses to Question 6 - the pilot’s subjective 
rating on their ability to safely complete the approach and landing - are broken down by lighting configuration and 
visibility. When broken out by approach lighting, the effect of improved lighting shows considerable (but not 
statistically significant) trends toward Ratings of 1 (“Excellent”). Conversely, VFR ratings for 1200 ft RVR average 
slightly below a rating of “Good.” For this condition, the pilot ratings ranged from 1 (“Excellent”) to 4 (“Almost 
Acceptable”). These data indicate that 1200 ft RVR approaches, even with only VFR lighting, are not “unacceptable” to 
the EPs and not significantly different from an ALSF-2 in the same visibility. These subjective results may suggest that 
visibility is the primary determinant of whether the approach can be completed to landing safely - not the lighting 
system. 

3.5. Non-Normals 

Non-normals were injected into the test unbeknownst to the evaluation pilots. The non-normals were four database 
integrity monitoring scenarios (lateral and altitude navigation position errors), two altimetry failure scenarios and one 
SV obstacle placement error scenario. The analyses of the altitude and SV obstacle placement error results are too 
lengthy for proper treatment herein. They will be the subject of another paper. 

Each pilot experienced two runs with a lateral navigation position error (40 meter right on breakout), each occurring with 
the same display location (head-up or head-down), but with one evaluation using the baseline information content and 
the other using the SVS information content. For all pilots, 43 landings were made out of 46 approaches (93%) despite 
the lateral position error. Of the three missed approaches flown, a go-around occurred for one Baseline HDD run, one 
SVS HDD run, and one SVS HUD run. With the Baseline HDD run, the pilot called “go-around” when he didn’t think 
he had enough time to recover from the error. This same pilot also went around with the SVS HDD run. On the SVS 
HUD missed approach condition, the pilot called “go-around” and then post-run, commented that there must have been a 
lateral shift. 
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The lateral navigational errors were flown with 2400 ft RVR and using the MALSR ALS and a 200 ft DA. The missed 
approach rate of 7% with these factors was identical to the nominal missed approach rate in these conditions, so the 
presence or absence of a 40 meter error did not affect the missed approach rate. 


Response to Q. 6 



□ ALSF-2 
■ MALSR 

□ VFR 


Figure 12. Pilot Mean Ratings of Their Ability to Complete the Approach and Land Safely. 


3.6. Recommendations / Future Work 

Numerous suggested improvements were identified. For instance, the pilots suggested that an alternative declutter 
strategy be explored on the HUD. Instead of removing all the symbology, several of the pilots suggesting displaying a 
reduced symbology set as they still wanted critical flight path information while completing the approach and landing. 
In addition, the SVS Head-down display evaluations used a 50 degree field-of-view. Less display minification should be 
explored to evaluate if improvements in performance using SVS HDD concepts can be obtained. 

4. CONCLUSIONS 

An experiment was conducted to identify the influence of cockpit display and airport lighting information and where the 
cockpit display information is provided to the pilot (e.g., on a Head-Up or Head-Down Display) on landing minima. A 
Global Positioning System / Wide Area Augmentation System (GPS/WAAS) approach was simulated to provide precise 
approach guidance instead of an instrument landing system (ILS), and the “operational considerations” evaluated under 
this effort included reduced visibility, decision altitudes, and airport equipage requirements, such as approach lighting 
systems, for SVS-equipped aircraft. Subjective results from the present study suggest that synthetic vision (SV) imagery 
on both head-up and head-down displays (compared to conventional displays without SV) may offer benefits in situation 
awareness (SA); workload; and approach and landing performance in the visibility levels, approach lighting systems, and 
decision altitudes tested. 

Subjective data indicates that significant improvements in SA can be provided by the integration of synthetic vision 
imagery with conventional flight symbology on both head-up and head-down primary flight displays. Both post- test and 
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post-run subjective measures indicate that the SV HUD provided the greatest SA among the 4 concepts tested. 
Subjectively, pilots felt that the SV imagery improved flight path control performance because the database imagery in 
the background provided stronger roll reference visual cues than the baseline condition (no SV imagery). Pilots’ post- 
run SA ratings showed no significant differences between the 2400 ft and 1800 ft RVR or between the three approach 
lighting systems. 

There were no pilot workload penalties found by increasing the “informational complexity” with the addition of SV 
imagery on either the head-up or head-down primary flight displays. In fact, pilot ratings indicated significant 
reductions in workload when SV imagery was used on the HUD compared to the Baseline HUD condition (flight 
symbology only). No significant differences were found between the 2400 ft and 1800 ft RVR levels for the pilot 
workload ratings. Similar to the SA ratings, the pilots’ post-run workload ratings showed no significant differences for 
the three approach lighting systems - VFR, MALSR, and ALSF-2 - used in this study. 

SA, workload and approach assessment ratings from both pilot groups (transport, FAA) showed similar trends in the 
benefits of SVS versus conventional symbology and the benefits of HUDs versus HDDs, but they differed in the 
magnitude (mean ratings) of these perceived benefits (higher SA, lower workload, more easily finding the runway and 
touchdown zone references, etc.). 

The ability of the flight crew to handle non-normal flight situations was explored. When presented with a 40 meter 
lateral positioning error, the rate at which the pilot’s executed a missed approach was identical to that without the 
position error. 
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